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Abstract 

The study used data from the Third International Mathematics and Science Study (TIMSS) 
to determine trends in calculator use among Population 2 (13-year-olds) students in Japan, the 
United States, and Portugal. While relatively-high levels of calculator use were observed for the 
U.S. and Portugal, virtually no calculator use was found for the Japanese sample. Hierarchical 
Linear Model analysis determined a significant negative relationship between students' frequency 
of calculator use and student performance in Japan; no significant relationship was detected for the 
U.S. and Portuguese samples. U.S. student achievement was positively associated with each of 
the five reported ways in which calculators are used, however a significant negative relationship 
was found between student performance and Japanese students' use of calculators on tests. 
Plausible explanations are explored. 
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A Comparison of Calculator Use in Eighth-Grade 
Mathematics Classrooms in the United States, Japan, and Portugal: 

Results From the Third International Mathematics and Science Study 

The Third International Mathematics and Science Study (TIMSS), conducted in 1994- 
1995, undertook to survey international school children and their teachers to determine the scope of 
several questions concerning educators. Among these questions were (1) the role of technology in 
teaching and learning mathematics and science, and (2) international variation in mathematics and 
science curricula. Three populations of students and their teachers were surveyed in 46 educational 
systems internationally. The subgroups of testing focused on 9 year-olds (Population 1), 13 year- 
olds (Population 2), and students in the last year of secondary school (Population 3). Data from 
these groups correspond to grades 4, 8, and 12 in most educational systems. The focus of 
research in this study was on Population 2 for two reasons: The researchers were highly interested 
in the use of technology by middle school mathematics students, and in the practices of their 
mathematics teachers. 



Background 

Recommendations of the National Council of Teachers of Mathematics 

In recent years, the advent of calculator technology has influenced the teaching of 
mathematics in a profound way in the United States (Demana & Waits, Dunham & Dick, 1994; 
1990; Fey & Good, 1985). The National Council of Teachers of Mathematics (NCTM) Curriculum 
and Evaluation Standards for School Mathematics (1989) has advocated that all students have 
access to calculators at all times and at all levels. The document includes the following comments 
concerning calculators: 

• The new technology not only made calculations and graphing easier; it has changed the 
very nature of the problems important to mathematics and the methods mathematicians 

use to investigate them; 
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• Calculators and computers for users of mathematics, like word processors for writers, are 

tools that simplify, but do not accomplish the work at hand; 

• The availability of calculators does not eliminate the need for students to learn algorithms. 

(NCTM, 1989, pg. 8) 

From the E rpfessional Standards for Teaching Mathematics (NCTM, 1991, p. 1 ), The 
Council recommends that teachers become more proficient in “using, and helping students use 
technology and other tools to pursue mathematical investigations.” The document (NCTM, 1991, 
p. 134) also states; “Technology changes the nature and emphasis of the content of mathematics as 
well as the pedagogical strategies used to teach mathematics. With the introduction of technology, 
it is possible to de-emphasize algorithmic skills; the resulting void may be filled by an increased 
emphasis on the development of mathematical concepts.” Notwithstanding the recommendations 
of the National Council of Teachers of Mathematics, the degree to which the Standards documents 
have been implemented has largely been undocumented. 

Research on Calculator T Ise 

There are several research findings which document the benefits of calculator use in the 
classroom. Campbell and Stewart (1993) found that calculators aid in problem solving, number 
sense, and understanding of arithmetic operations. Student confidence, enthusiasm and self- 
concept at all levels were increased by calculator usage (Campbell & Stewart, 1993; Hembree & 
Dessart, 1986). Suydam (1987) noted that over 100 studies reported that the use of calculators (a) 
promotes achievement, (b) improves problem-solving skills, and (c) increases understanding of 
mathematical ideas. Hembree and Dessart (1986) gathered information from 79 research reports 
and found that at every grade level from kindergarten through 12 (except Grade 4) that the use of 
calculators can improve the average student’s paper-and-pencil skills, both in basic operations and 
in problem solving. Suydam (1982) reported that no evidence had been found that elementary 
students become calculator dependent. Dunham (1993) found that nearly always, students taught 
with calculators (but tested without technology) had computation achievement scores as high or 
higher than those taught without technology. More recently, Dunham (1996) reported results from 
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14 dissertations, five journal articles, and seven conference papers; she found research continuing 
to show positive results or no significant difference for technology users in the classroom. 

Purpose of the Study 

Since the aforementioned research findings indicate positive results for calculators usage, 
the authors of the present study sought to investigate the major trends and perceived use of 
calculators by middle school students. The study used TIMSS data on calculator usage for 
Population 2 collected from both students and their teachers to ascertain how often calculators are 
used and the ways in which they are used. The present study reports the major trends in calculator 
use in Japan, United States and Portugal and relates calculator use to students’ mathematics 
achievement on TIMSS. 



Methodology 

The process of estimating each country’s score from the sample of students who 
participated in TIMSS produced only an estimate of the nation’s real score. This margin of error is 
expressed as an interval around each country’s estimated score. Because the precise score cannot 
be determined with perfect accuracy, nations were grouped into broad bands according to whether 
their performance was significantly higher than, not significantly different than, or significantly 
lower than the U.S.. Table 1 represents Nations’ Average Mathematics Performance Score 
Compared to the U.S.. 



(Insert Table 1 About Here) 



Selection of Nations 

In order to make international comparisons on students’ calculator use, Japan was selected 
from the group of 20 nations scoring significantly higher than the U.S., and Portugal was selected 
from the group of seven nations scoring significantly lower than the U.S.. Japan was chosen for 
its tradition of excellence in mathematics achievement, and because it was the only country from 
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the set of nations scoring significanUy higher than the U.S. for which qualitative data was 
available. Portugal, a European nation, was chosen after several of the nations scoring 
SignificanUy lower than the U.S. failed to meet international guidelines for TIMSS, and one other 
nation from the group excluded more than 10% of its population from testing. 

Data Analysis 

This study sought to ascertain to what degree calculator use is associated with American, 
Portuguese and Japanese eighth graders' mathematics score on TIMSS. As a statistical method, 
classical approaches, such as OLS, were avoided because typical education data has a complicated 
sample design that often violates the assumption of independence, namely, that sampled cases are 
independently drawn from the population. In TIMSS, students from the same mathematics 
classroom are sampled from each school. Because students in each school attend the school for the 
similar reasons and because they share the similar experiences, the cases that are sampled from the 
same schools are not independent from each other, leading to the cluster effect problem and 
underestimation of standard errors. 

Hierarchical Unear Model analysis (HLM) (Bryk, & Raudenbush, 1992) avoids the cluster 
effect problem by incorporating into the model a unique random effect for the group unit in which 
individuals are nested. In this application. Level 1 represents students and Level 2 represents 
schools. Intuitively, HLM executes a regression in each school and determines a set of coefficients 
that vaiy across group units. For example, schools have different intercepts; likewise, other 
coefficients that express association between predictors and the dependent variable also may vary 
across group units. These randomly varying Level 1 coefficients become outcomes at Level 2, 
which are then modeled by Level 2 predictors. Thus, HLM yields a decomposition of total 
variance into variances specific to the student and school levels. In this application, only the 
intercept will be set random because there were no expectations as to how the independent 
variables would vary across schools. See Appendix A for detailed specifications of HLM models 




employed in this application. 
Variable.s 
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Appendix B lists all the variables employed in the HLM analysis. The dependent variable 
was each nation’s average mathematics score, which were prepared to be internationally 
comparable. Because every student was not tested on the same items, the Item Response Theory 
(IRT) was employed to estimate a range of plausible values, rather than one single score. Five 
plausible values were used in this analysis; they contain not only the information about a student's 
proficiency but also the uncertainty or measurement imprecision that come with the measurement. 
Five regression models were run for each plausible value and from the five sets of estimates, the 
average was taken as the final result. 

One student-level independent variable, CALCLASS, was students' reports of frequency 
of calculator use during mathematics lessons. Student responses ranged from 0 (rare) to 3 (almost 
always). A second set of variables came from teacher surveys. Teachers were asked, "How often 
do students in your mathematics class use calculators for the following activities? Checking 
answers; Tests and exams; Routine computation; Solving complex problems; Exploring number 
concepts." Teachers’ responses were recoded to range from 0 (never) to 3 (almost every day). 
This set of five variables (CTffiCK, ROUTINE, TEST, SOLVE, and EXPLORE) from teacher 
surveys were entered into the HLM model at Level 2 because they represent characteristics of 
teachers rather than students (see Appendix A, note 3). Control variables included student’s 
gender, their use of extra instructions outside classrooms, their expected educational level to 
complete, and the types of communities that the schools were located. 

Limitations 

Ideally, there were certainly other important variables that should have been incorporated 
into the analysis. However, because the Japanese sample lacked students' private information, 
controls for home environment or ethnicity were not considered here. Even when the information 
was available for the U.S. and Portugal, it was not included so that the three nations could be 
contrasted using the same set of variables. Exclusion of such variables may be a source of model 
misspecification, and this is recognized as a possible weakness of the study. 
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Results 

This section is organized into two parts. First, descriptive statistics on calculator use is 
offered for each nation; data from student surveys precedes data from teacher surveys. Second, 
results of the HLM analysis are disclosed. More specifically, the regression analysis attempts to 
link student performance to frequency of calculator use and the ways in which it was used. The 
section concludes with a summary of research findings. 

Descriptive Analysis of Calculator Use 

S tudents’ reports on frequency of calculator use . Students’ use of calculators in 
mathematics lessons varied substantially by nation. As depicted in Table 2, nearly 68% of U.S. 
eighth-grade students reported they use calculators “almost always” or “pretty often.” A nearly 
identical portion, 69%, of Population 2 students in Portugal reported such regular calculator use. 
By way of contrast, calculator use among Japan’s eighth-grade students was profoundly low. In 
particular, less than 4% of Population 2 students reported they use calculators “almost always” or 
“pretty often,” and nearly three out of four Japanese students indicated they “never” use calculators 
during mathematics lessons. Of all nation’s participating in TIMSS, only Korea’s Population 2 
students reported less calculator use during instruction (Beaton, Mullis, Martin, Gonzalez, Kelly, 
& Smith, 1996). 



(Insert Table 2 about here) 

Teachers’ reports on w avs in which students’ use calculators. An analysis of teachers’ 
responses to questions on calculator usage in Population 2 classrooms revealed distinctive trends 
across the three nations. With respect to U.S. classrooms, regular use of calculators during 
instruction was reported by Population 2 teachers. As depicted in Table 3, U.S. students most 
commonly utilized calculators to solve complex problems (2.17) and to perform routine 
computations (1.96). Other reported uses of the calculator included exploring number concepts 

(1.71), and checking answers (1.52). Of the five reported ways in which calculators are used, 
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American students were least likely to use calculators during tests and exams, although the mean 
response of 1.42 suggests that many students are given access to such technology during test 
conditions. 

With respect to Portuguese students, calculator use was not statistically different than in 
U.S. mathematics classrooms. As shown in Table 3, Portuguese students most commonly used 
calculators to check answers (2.20) and to perform routine computations (2.12). They were 
somewhat less likely to use calculators to solve complex problems (1.89) and to explore number 
concepts (1.63) and like their American counterparts, they were least likely (1.40) to use 
calculators during testing conditions. 



Calculator use in Japan’s Population 2 classrooms was markedly different than both the 
U.S. and Portugal. As shown in Table 3, Japanese teachers reported little or no calculator use by 



the five means was significantly higher (p < .0001) than the corresponding mean for the U.S., and 
the same was true with respect to Portugal. In essence, with means ranging from 0.09 to 0.16, 
Japanese teachers indicate negligible use of calculators in eighth-grade mathematics and this is 
particularly true during test conditions. 

Results of the HLM Analysis 

Relating freauencv of calculator use to student performance . In order to model the effect of 
predictors such as calculator use, it is necessary to first establish the presence of substantial 
variation in the outcome variable, namely student performance. In Tables 4a-c, Model 1 is an 
analysis-of- variance model with no predictor. The intercept represents each nation’s average 
achievement score around which variation within and between schools is observed. The American 
sample showed a large proportion of variance between schools (36%), while for the Portuguese 
and Japanese samples, it was relatively small, 1 and 8%, respectively (see Appendix A, note 



(Insert Table 3 about here) 



students in each of the five aforementioned ways. A chi-square analysis determined that each of 
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4). This result implies that the school-level effect of calculator use will be harder to be detected in 
the Portuguese and Japanese samples than the American sample because there was not much level- 
2 variation to model in the first place. 

In Model 4, the key variable, students’ reported use of calculator during instruction, was 
considered at Level 1. As depicted in Table 4, the U.S. model suggests that the direction of effect 
was positive but not significant and the Portugal model shows the opposite, a negative effect that 
was not significant. The Japanese model, however, shows clear significance ( p < .001) and it 
was highly negative. In particular, a one unit increase in students’ reported use of calculators 
during instruction was associated with a 21-point decrease in student performance scores. The 
reduction of Level 1 variance was 1.4% when compared with a base Model 3 (see Appendix A 
note 5 for comparison of variance across models). This small reduction in variance makes sense 
because Japanese students report that they rarely use calculators in classrooms. 

(Insert Tables 4a, 4b, and 4c about here) 
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Relating wavs in which calculators are used to student performance . In Tables 5a-c, Model 
5 to Model 9 concern the uses of calculators for checking answers (CHECK), tests and exams 
(TEST), routine computations (ROUTINE), solving complex problems (SOLVE), and exploring 
number concepts (EXPLORE). As represented in Tables 5a-c, most of the large effects came from 
the U.S. sample. More specifically, each of the five reported ways in which calculators are used 
by American students was strongly associated (p < .(X)l) with student achievement. Each one unit 
increase in the use of calculators to check answers resulted in a 15.3-point increase in student 
performance; each incremental increase in calculator use on tests and exams was associated with a 
14.7-point increase, and an 1 1.2-point increase was observed as calculator use to perform routine 
computations increased incrementally. Even greater coefficients were determined for calculator use 
to solve complex problems and to explore number concepts, with a 20.1 -point and 18.4-point 
increase in performance, respectively. The reduction of Level 2 variance compared with an earlier 
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model, Model 3, was substantial. The most reduction of Level 2 vjuiance was observed with 
Model 8 (17%) and the least reduction was found with Model 7 (6%), the latter of which was still 
large. 



(Insert Tables 5a, 5b, and 5c about here) 

In contrast, Japanese and Portuguese teachers’ reports on ways in which calculators are 
used were not strongly associated with achievement. One particular item, TEST, from the 
Japanese sample a showed strong negative effect More precisely, for each one unit increase in 
this variable, there was a 20-point reduction in the school’s average test score. However, Level 2 
variance reduction compared to Model 3 was negligible. Other items showed no significance; such 
a result is understandable given that calculators are rarely used in Japanese mathematics 
classrooms. Further, as Model 1 suggested, there was not much variance at school-levels, hence, 
it was expected that Level 2 predictors would not show strong association with the outcome 
variables. The same applies to Portugal’s sample. No significant coefficient for calculator use was 
observed in the case of Portugal. 

Summary of Research Findings 

Students’ reports on frequency of calculator use was positively related with achievement in 
the U.S. but the coefficient size was not significantly large. It was associated negatively in 
Portugal’s sample, but only insignificantiy. Only for the Japanese sample was a significant 
negative coefficient observed. With respect to teachers’ reported ways in which calculators are 
used (checking answers, test/exams, performing routine computations, solving complex problems, 
and exploring number concepts), the American sample showed significantly large positive 
coefficients in all items. By way of contrast, the Japanese seunple showed a significantiy negative 
relationship only between student performance and their use of calculators on tests. No 
relationship between calculator use and student performance was detected in Portugal’s sample. 
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Discussion 

Data from both student and teacher surveys confirm that calculator use in eighth-grade 
classrooms varies substantially across nations. Perhaps most intriguing is the virtual absence of 
calculator use in Japanese eighth-grade mathematics classrooms, particularly given Japan's 
technologically-advanced society and its tradition of excellence in mathematics education. 
Moreover, this key research finding is even more remarkable when considering the ever-expanding 
body of research (e. g., Campbell & Stewart, 1993; Hembree & Dessart, 1986; Suydam, 1987) 
documenting the benefits of calculator use in fostering students' understanding of mathematical 
ideas. 

A second important finding is the relatively-high degree of calculator use by U.S. eighth- 
grade students. Whether or not American mathematics teachers are cognizant of standards calling 
for the provision of calculator technology, the results of this study suggest that most students have 
access such technology during instruction and under testing conditions. Indeed, only 10% of U.S. 
students comprising Population 2 indicated that they “never” use a calculator during mathematics 
lessons. 

In addition having access to calculators during instruction, results indicate that U.S. 
students are afforded opportunities to use calculators in nonroutine ways, including the exploration 
of number concepts and solving complex problems. It is interesting to note how U.S. student 
achievement was related to specific calculator uses during instruction. Activities that stress higher- 
order mentality, arguably to solve complex problems (SOLVE) and to explore number concepts 
(EXPLORE), showed the largest effect size, while more practical activities — to check answers 
(CHECK), on tests and exams (TEST), and to perform routine computations (ROUTINE) — 
showed smaller effect size. These results suggest that in the U.S. the positive effect of calculator 
use is particularly salient with activities that require higher-order thinking. Moreover, such results 
lend credence to the notion that calculators can foster students’ understanding of mathematical 
ideas. 
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The stark contrast in technology practices in Japan and the United States precipitates a 
search for possible explanations. That is, what factor(s) would preclude Japanese eighth-grade 
students from using a calculator during instruction or on mathematics exams? 

Discounting Curricular Differences 

It may seem plausible to attribute the minimal use of calculators in Japan to its mathematics 
curricula; that is, one may conjecture that Japanese eighth-graders are studying mathematics content 
that simply does not lend itself to hand-held calculators. If geometric concepts, for example, are 
the focus of study in eighth-grade, then Japanese students may find little or no use for such 
technology. But is calculator use in Japan and the U.S. attibutable curricular differences? 
Examination of each nation’s mathematics curriculum for eighth-grade students may yield answers. 
One caveat, however, is warranted. To determine the effect of particular curricular activities on 
student achievement, longitudinal data is needed, but not offered by TIMSS. Moreover, 
curriculum data from TIMSS was not available at press time. Notwithstanding this lack of data, it 
is possible to surmise Japan's implemented curriculum by examining its intended curriculum. 

Japan and Portugal represent two of the 29 nations in TIMSS whose mathematics 
curriculum is set at the national level. In Japan, the National Ministry of Education specifies one 
set of curriculum guidelines that specifies the number of instructional hours for each mathematics 
topic, and approves textbooks published by six commercial publishers. The national mathematics 
curriculum in Japan is focused. In particular, only three objectives for eighth-grade students are 
identified: 

1) To help students develop their abilities to compute and transform algebraic expressions 
using letter symbols according to their purposes, and to help them understand linear 
inequalities and simultaneous equations, and to foster their abilities to use them. 

2) To help students deepen their understanding of the properties of the fundamental figures 
in a plane, and thereby understand the significance and methods of mathematical 
inference with reference to consideration of the properties of figures, and to foster their 
abilities to precisely represent the process of inference. 
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3) To help students further deepen the way of viewing and thinking variation and 

correspondence and understand the characteristics of linear functions, and foster their 
abilities to use them. Furthermore, to help students adequately represent numbers 
according to their purposes and develop their abilities to grasp the tendencies of 
statistical phenomena. 

Although Japan’s integrated mathematics curriculum does devote some degree of attention to 
geometry, it also focuses heavily on algebraic concepts as well as on statistical and algebraic 
reasoning, all of which provide abundant opportunities for calculator use. Consequently, the 
minimal use of calculators in Japan does not appear to be related to its mathematics curriculum. 

In contrast to Japan and Portugal, the U.S. mathematics curriculum is determined at local 
levels and often follows broad guidelines as set by each of the 50 states. Because the mathematics 
curriculum is not similar among or even within the educational systems surveyed, it is difficult to 
get a clear sense of a typical eighth-grade mathematics curriculum, particularly since only 28% of 
U.S. students use one of the five most-conunonly used textbooks. Nevertheless, national 
recommendations (NCTM, 1989), on which state frameworks and local school district guidelines 
are largely based, place increased emphasis on developing number sense, identifying and using 
functional relationships, using statistical methods, and using appropriate technology for 
computation and exploration. Such recommendations represent a departure from curricular 
practices of the 1980’s when the U.S. junior high school curriculum was primarily devoted to 
developing students' computational skills (McKnight, Crosswhite, Dossey, Kifer, Swafford, 
Travers, & Cooney, 1987). Whether U.S. students are subjected to a reform-based mathematics 
curriculum or one that is more arithmetic-latent, their curriculum affords them numerous 
opportunities to use calculators during instruction. 

Cultural Differences 

Because differences in American and Japanese technology practices do not appear to be a 
related to their respective mathematics curricula, alternative explanations must be sought. Are such 
polar educational practices more attributable to cultural differences? In Japan, educators and 
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students may not be enthusiastic about the use of calculator. A principle reason for this is that an 
important emphasis of education in Japanese junior high schools is preparation for the high school 
entrance examination. In this examination, the use of calculators is not allowed; thus, in preparing 
students for this exam, teachers may put a stress on a student’s ability to perform mathematical 
operations without the use of a calculator. Generally speaking, use of calculators in mathematics 
classrooms is associated with "cheating." 

Further, a general apathy towards the use of technology in education comes from a cultural 
ethos that downplays technology in Japan. Education is associated with discipline, compliance to 
the authority, and socialization, rather than creative processes towards the development of 
individuality and skills (Rohlen, 1983). As a result, the use of both calculator and computer 
technology may be perceived as a deviation from the cultural ideals of education. 

Agenda for Future Research 

Results of the present study revealed calculator use and its association with student 
achievement varies greatiy between the nations of Japan, the United States, and Portugal. 
Although such relationships were ascertained for Population 2, data from Population 1 (9-year- 
olds) and Population 3 (17-year-olds) remain imexplored. Accordingly, it is the intent of the 
research team to analyze both teacher and student data from Populations 1 and 3 in order to 
determine possible links between calculator use and student performance. 
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Table 1 



Nations Average M athematics Performance Compared to the United States 



NATIONS WITH 
AVERAGES 
SIGNIFICANTLY 
HIGHER THAN THE 
UNITED STATES 


NATIONS WITH 
AVERAGE SCORES 
NOT SIGNIFICANTLY 
DIFFERENT FROM 
THE UNITED STATES 


NATIONS WITH 
AVERAGE SCORES 
SIGNIFICANTLY 
LOWER THAN THE 
UNITED STATES 


NATION AVERAGE 


NATION AVERAGE 


NATION AVERAGE 


Singapore 


643 


(Thailand) 


522 


Lithuania* 


477 


Korea 


607 


(Israel)* 


522 


Cyprus 


474 


Japan 


605 


(Germany)** 


509 


Portugal 


454 


Hong Kong 


588 


New Zealand 


508 


Iran, Islamic Republic 


428 


Belgium-Flemish* 


565 


England 


506 


(Kuwait) 


392 


Czech Republic 


565 


Norway 


503 


(Colombia) 


385 


Slovak Repulbic 


547 


(Denmark) 


502 


(South Africa) 


354 


Switzerland* 


545 


United States* 


500 






(Netherlands) 


541 


(Scotland) 


498 






(Slovenia) 


541 


Latvia (LSS)* 


493 






(Bulgaria) 


540 


Spain 


487 






(Austria) 


539 


Iceland 


487 






France 


538 


(Greece) 


484 






Hungary 


537 


(Romania) 


482 






Russian Federation 


535 










(Australia) 


530 










Ireland 


527 










Canada 


527 










(Belgium-French) 


526 










Sweden 


519 











Hfltgs. Nations not meeting international guidelines are shown in parentheses. Nations in which 
more than 10 percent of the population was excluded from testing are shown with a *. Latvia is 
designated LSS because only Latvian-speaking schools were tested, which represents less than 65 
percent of the population. Nations in which a participation rate of 75 percent of the schools and 
students combined was achieved only after replacements for refusals were substituted, are shown 
with a •. The country average for Sweden may appear to be out of place; however, statistically, its 
placement is correct. 
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Table 2 

Students' Reports o n Frequency of Using Calculators During Mathematics Lessons — Japan. 
U.S.. and Portugal 





Never 


Once in a While 


Prettv Often 


Almost Always 


Country 


Percent 

of 

Students 


Mean 

Achieve- 

ment 


Percent 

of 

Students 


Mean 

Achieve- 

ment 


Percent 

of 

Students 


Mean 

Achieve- 

ment 


Percent 

of 

Students 


Mean 

Achieve- 

ment 


Japan 


74.7 

(2.27) 


607.24 

(2.06) 


20.8 

(1.88) 


603.04 

(3.37) 


3.46 

(0.65) 


575.35 

(6.69) 


0.37 

(0.09) 


524.37 

(24.12) 


u.s. 


9.88 

(1.52) 


463.87 

(9.39) 


19.62 

C1.54) 


498.28 

(5.83) 


24.86 

(1.16) 


501.22 

(5.30) 


43.03 

(2.66) 


510.63 

(5.64) 


Portugal 


3.02 

(061) 


455.29 

(7.32) 


27.19 

(1.59) 


456.56 

(3.10) 


34.07 

(1.19) 


453.87 

(3.50) 


34.80 

(1.45) 


454.08 

(2.83) 



Note. Standard Errors are reported in parentheses. 
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Table 3 

Teachers' Reports on Wavs in Which Calculators are Used in Mathematics Lessons -- Japan. 
U.S.. and Portugal 





Checking 

Answers 


Tests 

and 

Exams 


Routine 

Computation 


Solving 

Complex 

Problems 


Exploring 

Number 

Concepts 


Country 


Mean Response 


Mean Response 


Mean Response 


Mean Response 


Mean Response 


Japan 


0.04 

(0.02) 


0.01 

(0.00) 


0.14 

(0.04) 


0.11 

(0.03) 


0.16 

(0.05) 


U.S. 


1.52 


1.42 


1.96 


2.17 


1.71 


(0.09) 


(0.11) 


(0.11) 


(0.09) 


(0.09) 


Portugal 


2.20 

(0.10) 


1.40 

(0.08) 


2.12 

(0.09) 


1.89 

(0.10) 


1.63 

(0.09) 



Note. Teachers responses were recoded in the following manner: 0 = Never or Hardly Ever, 
1 = Once or Twice Per Month, 2 = Once or Twice a Week, 3 = Almost Every Day. Standard 
Errors are reported in parentheses. 
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Appendix A: HLM Specification 
NOTE 1: Sample Weight in HLM analysis 

"At level- 1 (student level) cases should be weighted by the 'normalized' inverse of the 

student's selection probability given school is selected. In TIMSS, this can be obtained by first 

calculate the SUBTOTWT for student j of school i as following: 

SUBTOTWTij=TOTWGTij/SCHWGTi 

and then 'normalize' it to get the weight variable SUBRELWT: 

SUBRELWTij=SUBTOTWrij/sum (of SUBTOTWTij)/ni 

I 

here ni is the number of students selected in school i. SUBRELWT should be used at level- 1. 
But don't let HLM 'normalize' it since it has been normalized whithin each school. 

At level-2 (school level) cases should be weighted by the 'normalized" school weights. For 
TIMSS, simple use weight SCHWGT and let HLM normalize it." 

(FROM a hand-out provided in TIMSS seminar held in Maryland, November 1998, with a 
note "reference: D. Pfeffermann et al. To appear JRSS(B?) 1998) 

NOTE 2: CENTERING 

HLM program is set such that all level- 1 variables will be centered around the sample grand 
mean when they are actually regressed against the outcome variable. This gives the intercept a 
substantive meaning for interpretation (Bryk & Raudenbush, 1992, p.25). 

NOTE 3: Problem of LEVELS 

In this application, level-2 are classroom, as well as schools. Ideally, we would like to have 
students at level-1, classrooms at level-2, and schools at level-3; however, only one classroom 
in Japan and Portugal and two classrooms in the US are surveyed. Technically, one needs at 
least (n of predictors + 1) cases at each group unit in HLM. Because the number of classroom 

in each school is too small, we decided not to consider classrooms as distinct level and treat 
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classrooms and schools at the same level-2. Another problem in this format was the situation 
where Japanese and Portugal students are rightly considered to be nested within one 
classroom, but American students are nested within two classrooms. To represent American 
teachers' calculator use at school level, I used the average of two classrooms sampled. 
Otherwise, the variable concerning teachers' calculator use will go to different levels for 
American sample and for Japanese and Portugal samples, making our interpretation difficult. 
We admit our "cheating levels" and consider it as our weakness, although any HLM application 
of TIMSS will face the same sets of such decisions. 

NOTE 4: Intraclass correlation 

Intraclass correlation tells us what proportion of variance is at level-1 and level-2. It is 
acquired by dividing the variance that comes from one level by the total variance (i.e., level- 
1 variance/level- lvariance+level-2 variance, or level-2 variance/level-lvariance+level-2 
variance). Variance is not an easy statistical construct to compare across coimtries because it 
may have different shapes; however, we believe that the difference between US and the other 
two appear substantial enough. 

NOTE 5: Comparison of Variance across models 

HLM model decomposes the total variance into level-1 variance and level-2 variance. Level-1 
variance of one model is comparable to level-1 variance of another model only when both 
models have the same level-2 predictors. Likewise, level-2 variance of one model is only 
comparable to level-2 variance of another model when both models have the same level-1 
specifications. For example, one cannot compare the level- 1 variance of Model 1 and Model2 
because level-2 specification of these models are different. Level-2 variance is comparable 
between Model-1 and Model-2 because level-1 specification of these models are the same. 
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Appendix B: Variables 



KEY VARIABLES- 

Dependent variable 

Mathscorei five plausible values, BSMPVOl-05 in TIMSS acronym. 

Independent variable 

LEVEL-1 (student level) 

CALCLAS: student-reported use of calculator, 0=never, l=once in a while, 
2=pretty often, 3=almost always. 

LEVEL-2 (teacher/classroom/school level) 

"How often do students in your mathematics class use calculators for the following 
activities? Range: 0=never or almost never, l=some lessons, 2=most lessons, 
3=every lesson. 

CHECK: Checking answers. 

TESTD: Tests and exams. 

ROUT: Routine computation. 

SOLV: Solving complex problems. 

EXPL: Exploring number concepts. 

CONTROL VARTART F.S - 

LEVEL- 1 (student level) 

GIRL : a dummy variable of 1 if student are female and 0 if male. 

TMJUKU : the amount of time that students take extra lessons before or after 
schools. 0=no time, l=less than 1 hour, 2=1 to 2 hours, 3=3 to 4 hours, 
4=more than 4 hours. 
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SED: the maximum level of education students are hopeing to complete. 0=1 don't 
know, l=finish elementary school, 2=finish some high school, 3=some 
vocational/technical education after high school, 4=some community college, 
college, or university courses, 5=complete a bachelor's degree at a college or 
university. 

LEVEL-2 (teacher/classroom/school level) 

RURAL: schools located in rural or isolated areas, 

A dummy variable of 1 if yes, else 0. 

URBAN: schools located in rural or isolated areas. 

A dummy variable of 1 if yes, else 0. 

SUBURB:schools located in rural or isolated areas. 

A dummy variable of 1 if yes, else 0. 

LOCMISS: schools whose location is missing. 

A dummy variable of 1 if yes, else 0. 
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Appendix C: Level 1 and 2 Descriptive Statistics 



USA 

LEVEL-1 DESCRIPTIVE STATISTICS 



VARIABLE NAME 


N 


MEAN 


SD 


MINXMOM 


MAXIMUM 


BSMFVOl 


7087 


490.88 


92.22 


172.03 


816.67 


BSMFV02 


7087 


490.68 


92.37 


172.36 


816.31 


BSMPV03 


7087 


490.72 


92.22 


168.23 


815.69 


BSMFV04 


7087 


491.11 


92.59 


167.82 


816.09 


BSMFV05 


7087 


490.60 


92.94 


167.23 


816.98 


CALCHQME 


6985 


0.98 


0.15 


0.00 


1.00 


CALCLASS 


6892 


3.02 


1.03 


1.00 


4.00 


DGIRL 


7087 


0.50 


0.50 


0.00 


1.00 


mjUKU 


6762 


0.45 


0.74 


0.00 


4.00 


SED 


6682 


4.13 


2.29 


0.00 


6.00 


LIKEMTH 


6923 


2.83 


0.93 


1.00 


4.00 





LEyEL-2 


DESCRIPTIVE 


! STATISTICS 






VARIABLE NAME 


N 


MEAN 


SD 


MINIMUM 


MAXIMUM 


CHECK 


181 


2.09 


1.07 


0.00 


3.00 


TEST 


181 


1.47 


1.06 


0.00 


3.00 


ROUT 


181 


1.98 


1.09 


0.00 


3.00 


SQLV 


181 


2.12 


0.98 


0.00 


3.00 


EXPL 


181 


1.74 


1.03 


0.00 


3.00 


SCHWGT 


181 


1.00 


2.60 


0.00 


31.29 


RURAL 


181 


0.17 


0.38 


0.00 


1.00 


SUBURB 


181 


0.25 


0.43 


0.00 


1.00 


URBAN 


181 


0.43 


0.50 


0.00 


1.00 


LOCMISS 


181 


0.15 


0.36 


0.00 


1.00 









FORrUGAL 








LEVEL- 1 DESCRIPTIVE STATISTICS 






VARIABLE NAME 


N 


MEAN 


SD 


MINIMUM 


MAXIMUM 


BSMPVOl 


3391 


455.14 


64.56 


261.59 


671.89 


BSMPV02 


3391 


455.25 


63.11 


227.58 


671.68 


BSMFV03 


3391 


455.19 


63.94 


228.03 


671.32 


BSMFV04 


3391 


455.91 


63.55 


230.90 


671.55 


BSMPV05 


3391 


454.92 


63.59 


227.24 


672.10 


CALCHOME 


3374 


0.99 


0.08 


0.00 


1.00 


CALCLASS 


3357 


3.02 


0.86 


1.00 


4.00 


DGIRL 


3391 


0.49 


0.50 


0.00 


1.00 


TMJUKU 


3132 


0.49 


0.81 


0.00 


4.00 


SED 


2886 


3.69 


2.23 


0.00 


6.00 


LIKEMTH 


3366 


2.79 


0.85 


1.00 


4.00 





LEVEL-2 


DESCRIPTIVE 


STATISTICS 






VARIABLE NAME 


N 


MEAN 


SD 


MINIMUM 


MAXIMUM 


CHECK 


142 


2.37 


0.87 


0.00 


3.00 


TEST 


142 


1.42 


1.07 


0.00 


3.00 


ROUT 


142 


2.12 


1.13 


0.00 


3.00 


SQLV 


142 


1.88 


1.11 


0.00 


3.00 


EXPL 


142 


1.66 


1.12 


0.00 


3.00 


SCHWGT 


142 


1.00 


0.77 


0.30 


6.36 


RURAL 


142 


0.09 


0.29 


0.00 


1.00 


SUBURB 


142 


0.25 


0.44 


0.00 


1.00 


URBAN 


142 


0.59 


0.49 


0.00 


1.00 


LOCMISS 


142 


0.06 


0.24 


0.00 


1.00 
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JAPAN 

LEVEL-1 DESCRIPTIVE STATISTICS 



V2\RIAHLE NAME 


N 


MEAN 


SD 


HINIHUM 


MAXIMUM 


BSMFVOI 


5141 


603.16 


101.74 


244.70 


891.33 


BSMPV02 


5141 


603.37 


101.18 


244.95 


954.29 


BSMPV03 


5141 


604.12 


101.83 


240.22 


953.41 


BSMPV04 


5141 


604.25 


102.67 


239.89 


953.99 


BSMPV05 


5141 


604.46 


100.96 


239.46 


955.19 


CALCLASS 


5107 


1.29 


0.54 


1.00 


4.00 


DGIRL 


5141 


0.49 


0.50 


0.00 


1.00 


TMJUKU 


5096 


1.28 


1.09 


0.00 


4.00 


SBD 


5091 


3.79 


2.42 


0.00 


6.00 


LIKEMTH 


5095 


2.52 


0.81 


1.00 


4.00 





LEVEL-2 


DESCRIPTIVE 


STATISTICS 






VARIABLE NAME 


N 


MEAN 


SD 


MINIMUM 


MAXIMUM 


CHECK 


151 


0.04 


0.23 


0.00 


2.00 


TEST 


151 


0.01 


0.08 


0.00 


1.00 


ROUT 


151 


0.12 


0.38 


0.00 


2.00 


SQLV 


151 


0.10 


0.33 


0.00 


2.00 


EXPL 


151 


0.14 


0.48 


0.00 


3.00 


SCRWGT 


151 


1.00 


0.58 


0.63 


3.57 


RURAL 


151 


0.21 


0.41 


0.00 


1.00 


SUBURB 


151 


0.42 


0.49 


0.00 


1.00 


URBAN 


151 


0.36 


0.48 


0.00 


1.00 


LOCMISS 


151. 


0.01 


0.08 


0.00 


1.00 
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